Brand sonification

ABSTRACT

A mobile device comprising a software application configured to detect the sound of a product use event; provide a user reward using said software application in response to said detection; capture data relating to said product use event; and provide said captured data to a remote computer system for analysis.

FIELD OF THE INVENTION

The present invention relates to identifying sounds emitted by aconsumer product when activated, used or opened, and in particular tosounds emitted by the opening of packaging of fast moving consumer goodssuch as soft drinks.

BACKGROUND TO THE INVENTION

Background information on sound identification systems and methods canbe found in the applicant's PCT application WO2010/070314, which ishereby incorporated by reference in its entirety.

The term “brand sonification” describes the use of sounds (typicallynon-verbal sounds) to convey brand related information. Many brandstoday are associated with a specific sound, such that when a consumerhears the sound, they immediately think of the associated brand. Currentcommonly known examples include the Intel ® “Intel inside” jingle orMcDonald's Corporation's five-note “I'm lovin' it” jingle. However,brand sonification is not limited to sounds, jingles or themes that arespecially created for a brand, but also encompasses the sounds createdby branded goods when they are activated, used or opened. For example,the sounds made when a canned beverage is opened or when computerequipment is booted-up can also convey brand information. Such soundsmay make up a significant proportion of an advertising campaign, e.g.within a promotional video or radio advertisement.

The present applicant has recognized the need to be able to identifysounds created by such branded goods when they are used or activated.

SUMMARY OF THE INVENTION

As mentioned above, “brand sonification” is the characteristic soundthat a product emits when activated, used or opened. The product couldinclude, but not be limited to: food or beverages; home or personal careproducts in primary packaging which emits a sound on opening; computerequipment or software applications which emit or cause to emit a soundon power-up or activation; and any other product that emits a sound whenused or activated. For example, the opening of a pressurized beverage orthe opening of a packet of crisps/chips may generate particular soundsthat are characteristic of a particular brand.

Generally speaking, a system to detect the sound of a product use eventcomprises a microphone, processing unit, software, decision-makingprocess and possibly connectivity to other systems. The system has thecapability to detect the brand sonification distinct from other sounds,e.g. from background sounds associated with where the product is used(e.g. at work or in a park). Having detected the brand sonification, thesystem can initiate a further process which can either:

Provide a level of interactivity with the user of the product, includingbut not limited to: linking to on-line content; activating specificsoftware applications; initiating taking of photographs; issuing rewardsof some form to the user; etc.; or

Provide data to an information system (i.e. to the remote computersystem) which records or analyses data relating to product usage orconsumer behavior.

Thus, according to a first aspect of the invention, there is provided amobile device comprising a software application configured to: detectthe sound of a product use event; provide a user reward using saidsoftware application in response to said detection; capture datarelating to said product use event; and provide said captured data to aremote computer system for analysis.

Preferably, the mobile device is a mobile telephone, a smartphone, atablet computer etc. The sound is received by a microphone locatedwithin the mobile device. A software application running on the mobiledevice may be configured to interact with the microphone to detect thesound. A processing unit within the mobile device or hosted on ‘thecloud’ may perform the sound identification analysis. In particular, theprocessing unit determines whether the detected sound matches storedsound models.

In embodiments, the sound of a product use event is detected byalternative electronic goods such as a camera or a beverage and/or foodvending machine. This is described in more detail below.

In embodiments, the sound of the product use event comprises the soundof opening a package of the product. For example, the softwareapplication may be configured to distinguish between the sound of apressurized beverage can being opened and the sound of a pressurizedbeverage bottle being opened, and even between the sounds of differentsized bottles and cans being opened (e.g. between 500 ml, 1 l, 1.5 lbottles etc.).

Preferably, this may be the sound of a can ring-pull/tab opening eventon a can of pressurized beverage, and/or the sound of a screw captwisting and opening event on a beverage bottle. However, as mentionedabove, the sound could be associated with the opening of a bottledbeverage, the booting of a PC, the opening of packaging of a food item,the consuming of a crunchy food product etc.

In embodiments, the captured data is location data of the mobile devicewhen the sound is detected and/or date and time data linked to when thesound is detected. The location of the mobile device may be determinedusing a GPS-capability of the mobile device or other similar means. Thelocation and time/date information of the product use event may betransmitted to a remote computer system run by the owner of theidentified brand, in order to provide the brand owner with preciseinformation on the usage of their products. Such information may, forexample, enable a brand owner to determine that one of their products istypically used by consumers on weekdays at lunchtime.

In embodiments, upon detection of the sound, the user may be rewarded byaccess to exclusive media, such a video or music file. Additionally oralternatively, the user reward is a monetary reward such as a fixed orpercentage discount off a future purchase.

In embodiments, the user reward is only delivered when the softwareapplication has detected a specific number of product use events. Thismay incentivise the user to utilize the software application, and alsoprovides the brand owner with a fixed amount of usage data beforepermitting a reward to be delivered to the user.

In a related aspect of the invention there is provide a non-transitorydata carrier carrying processor control code for the above-mentionedsoftware application.

According to a further aspect of the invention, there is provided amethod of capturing product use data from users of a product, the methodcomprising: providing a mobile device with an app configured to identifyuse of said product from a sound of said use and to provide aninteractive user experience in response to said identification; andcapturing use data of said product during use of said app.

In embodiments, the method of capturing product use data comprisesmodifying the product to increase a distinctiveness of the productsound.

For example, in a particular embodiment, a can ring-pull/tab on a can ofpressurized beverage may be provided with ridges, grooves and/orindentations to increase the distinctiveness of the product sound as thering-pull is being activated to open the can.

In another embodiment, a screw cap on a beverage bottle is connected toa band (e.g. a tamper-evident band) by a plurality of breakable thinconnectors, and the screw cap, the band and/or the plurality of thinconnectors may be modified to increase the distinctiveness of theproduct sound when the screw cap is twisted and the thin connectors arebroken (to disconnect the screw cap from the tamper-evident band).

According to another aspect of the invention, there is provided a methodof capturing product use data from users of a product, the methodcomprising: providing a consumer electronic device having at least onemicrophone with access to audio analytic software configured to analyzeaudio data captured by said microphone to identify a sound associatedwith use of said product; using said consumer electronic device toidentify when said product is used from said sound of use of saidproduct; and capturing data relating to said use of said product inresponse to said identification.

Preferably, the method of capturing product use data comprises providingthe consumer electronic device with an interactive software applicationhaving access to audio analytic software, wherein the interactivesoftware application is configured to enable user interaction with theconsumer electronic device in response to the sound identification.

In particular embodiments, the audio analytic software is providedwithin the consumer electronic device. Thus, the consumer electronicdevice, e.g. a mobile telephone or tablet computer, is able to performthe sound identification itself.

Alternatively, the audio analytic software may be provided on a networkof remote servers hosted on the Internet (i.e. ‘the cloud’), and thecaptured audio data may be transmitted to the remote servers for theanalysis. This may be preferred in certain circumstances, such as whenthe consumer electronic device lacks the processing power to be able toperform the audio analysis/sound identification. Thus, the processor ofthe consumer device may be configured to receive the audio data andevent data and transmit the data to ‘the cloud’ for analysis.

In both embodiments (i.e. local or cloud-based processing), theinteractive software application has access to one or more sound modelsand the audio analytic software compares the captured audio data withthe one or more sound models to identify the sound.

Preferably, the one or more sound models are updated and improved usingthe audio data captured by the and each consumer electronic device usingthe audio analytic software.

In the case where the audio analytic software and the one or more soundmodels are provided on the network of remote servers, audio datacaptured from one or more consumer electronic devices is transmitted tothe remote servers to enable the one or more sound models to be updated.

Alternatively, in the case where the audio analytic software is providedwithin the consumer electronic device, the updated sound models aredelivered to the consumer electronic device such that the software canaccess the updated models locally (i.e. within the device).

In a related aspect of the invention, there is provided a can ofpressurized beverage having a ring-pull or push-tab in a top of the can,wherein one or both of said ring-pull/push-tab and said can top areconfigured to interact with one another on opening to generate adistinctive can-opening sound in addition to a sound made by breaking ofthe frangible seal of the tab and escape of pressurized gas from withinthe can.

In a further related aspect of the invention, there is provided amarketing data collection system comprising the above-described mobiledevice in combination with the above-mentioned remote computer system.

According to an aspect of the invention, there is provided a system foridentifying a sound associated with a product use event, the systemcomprising: non-volatile memory for storing one or more sound models andfor storing processor control code; a sound data input; a processorcoupled to said sound data input and to said stored processor controlcode, wherein said processor control code comprises code to: input, fromsaid sound data input, sample sound data for said sound associated withsaid product use event to be identified; input event data associatedwith said sound data input; compare said sample sound data with saidstored one or more sound models; identify a product associated with saidsample sound data; and deliver an interactive user experience associatedwith said identification to a user of said product.

As mentioned earlier, the sound of a product use event may be detectedby a mobile device but also by consumer electronic devices such as, butnot limited to, home or office PCs, video game consoles, digitalcamcorders, or digital cameras, or by commercial electronic devices suchas vending machines. Thus, in embodiments, the sound data is captured bya consumer electronic device or a vending machine, and wherein thecaptured sound is transmitted by said consumer electronic device or saidvending machine by a wired or wireless connection to the system forprocessing.

Many digital cameras have the capability to record moving videos withsound (i.e. they also have a microphone). Thus, a digital camera may beable to detect the sound of a product use event. In this case, thecomputational capability of the camera's processor may not be sufficientto perform the sound analysis. However, many digital cameras can bewired or wirelessly connected to other devices (e.g. PCs, laptops etc.)or have internet capability such that they can use mobile networks orwireless connections to share data with other devices (e.g. mobiledevices) or with social networking services. The audio analytic softwaremay be provided on a network of remote servers hosted on the Internet(i.e. ‘the cloud’) or may be provided on a further electronic device(e.g. a mobile device), and the audio data captured by the digitalcamera may be transmitted to the remote servers or the furtherelectronic device for the analysis. The camera may also transmitinformation about the time and date of the sound capture, which mayprovide a brand owner with useful information about when their productis being consumed/used. The ‘reward’ or interactive user experience maybe delivered to the user via email (e.g. using the email address usedwhen registering the product), by SMS/MMS to a mobile phone associatedwith the user of the camera, to the camera itself, to the user's socialnetwork profile, or by other means.

Vending machines typically dispense food items and beverages, but maydispense other types of consumer goods such as alcohol, cigarettes,hygiene products etc. Modern vending machines may have interactive touchscreens, video analytics to recognize the gender and age of the user,and cloud manageability of sales and inventory. Thus, vending machinesmay have or may be provided with microphones to enable the machines todetect the sound of a product use event. Users of the vending machinemay use a product (e.g. open a beverage container or snack packaging) assoon as the good has been dispensed by the machine, and the microphonecan detect the sound. As with the camera example above, the vendingmachine itself may not have the processing power to be able to performthe sound analysis itself, but may transmit the audio data to a furtherdevice or to ‘the cloud’ for analysis. The vending machine may alsotransmit data on the time and day of the sound detection, and dataidentifying the machine (which can help a brand owner determine thegeographical location of the machine). The ‘reward’ or interactive userexperience delivered to the user may in this case be a voucher or codefor a monetary or percentage discount off a future purchase, which mayincentivise the user to use the same vending machine in the future or topurchase the same product again. The voucher or code may be delivered tothe vending machine user via the display or interactive touch screen onthe vending machine, or by prompting the machine to print out avoucher/code for the user to take with them.

Accordingly, in a related aspect of the invention there is provided avending machine comprising a software application configured to: detectthe sound of a product use event; provide a user reward using saidsoftware application in response to said detection; capture datarelating to said product use event; and provide said captured data to aremote computer system for analysis.

The skilled person will understand that many other types of electronicgoods (both consumer goods and commercial) can be configured to detectthe sound of a product use event and to transmit the audio data to afurther device or to ‘the cloud’. The skilled person will furtherunderstand that features of embodiments and aspects of the inventiondescribed herein as incorporated into or associated with a mobile devicemay similarly be incorporated into or associated with another devicesuch as a vending machine, games console, or other device as describedabove.

The invention also provides processor control code for theabove-described systems and methods, in particular on a data carriersuch as a disk, CD- or DVD-ROM, programmed memory such as read-onlymemory (Firmware), or on a data carrier such as an optical or electricalsignal carrier. Code (and/or data) to implement embodiments of theinvention may comprise source, object or executable code in aconventional programming language (interpreted or compiled) such as C,or assembly code. As the skilled person will appreciate such code and/ordata may be distributed between a plurality of coupled components incommunication with one another.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is diagrammatically illustrated, by way of example, in theaccompanying drawings, in which:

FIG. 1 shows a process to detect brand sonification in an embodiment ofthe invention;

FIG. 2 shows a process to detect brand sonification via the internet inan embodiment of the invention;

FIG. 3 shows a process to detect brand sonification on a local computingdevice in an embodiment of the invention; and

FIG. 4a illustrates the mechanism of opening a typical pressurized canand FIG. 4b illustrates an example can modification to increase productsound distinctiveness.

DETAILED DESCRIPTION OF THE DRAWINGS 1. Brand Sonification

FIG. 1 shows a general process to detect brand sonification in anembodiment of the invention. A sound is generated by a product when itis used, activated or opened. For example, the opening of a pressurizedbeverage or the opening of a packet of crisps/chips may generateparticular sounds that are characteristic of a particular brand. Thesound is generated by an audio generation device of the product. Forinstance, on a pressurized beverage can, the pulling of a ring-pull orpull-tab on the lid can generates a sound as a scored part of the lidcomes away from the rest of the can lid. In another example, the pullingapart of a crisp/chip packet when opening may generate audio. In afurther example, the consuming of the crisps/chips within the packet mayin itself generate audio which is characteristic of a particular brand,e.g. the crunching sound.

In the illustrated embodiment, the brand sonification detection isperformed using a mobile device comprising a software application. Themobile device may be a mobile telephone, a smartphone, a tablet computeretc. The software application is configured to detect the sound, capturedata relating to detected sound, provide a user with a ‘reward’ andforward the captured data to a remote computer system for analysis—theseprocesses are described in more detail below.

Generally, the sound is received by a microphone located within themobile device via acoustic coupling (e.g. using an acoustic coupler orsimilar device within the mobile device). The software application maybe configured to activate when the microphone detects a sound. Themicrophone sends the signal to a processing unit. As is described belowin more detail, in particular embodiments, the processing unit withinthe mobile device may perform the sound identification analysis, whilein alternative embodiments, an external processing unit may perform theanalysis.

The processing unit and software (either within the mobile device orexternal to it) determine whether the received sound matches particularstored sound models. In general, the sound models are generated in twoways:

By reducing thousands of similar sounds into their constituent parts toenable a model to be generated for a particular class of sounds. This isachieved by collecting thousands of hours of audio recordings, in a widerange of environments using a wide variety of different recordingequipment. For example, for brand sonification, the audio recordings maybe of different products being used, activated or opened (e.g. beveragecontainers being opened, crisps/chips being eaten, software applicationsbeing initialized) in different environments (e.g. home, office, in apark, in a café etc.). This sound data allows product use to beidentified in the presence of a range of different background sounds.Received audio data is compared to the stored sound models to determineif the received audio data has constituent parts that match those of aparticular model.

By using a closed-loop system to update and improve existing soundmodels based on audio data received from users of mobile devices.Although the sound models may have been created using thousands of audiorecordings, the audio data may not represent all possible ways in whicha product may be used or the different environments the product may beused in. For example, the sound a pressurized beverage container makesmay depend on how the user holds the can or the specific position of theuser's fingers on the ring-pull. In a further example, the sounds ofsome consumables being used or opened may differ under differentpressures. Thus, the actual sounds generated by use of a product byusers can be used to help improve existing models.

More details on how sounds identification models are determined and hownew sounds are identified or matched to models can be found respectivelyin sections 3 and 4 below. If the processing unit establishes that thesound matches a known model, the received sound is considered a ‘productuse event’. Data associated with the event is then transmitted to afurther system or systems which is(are) located externally to the mobiledevice. The event data may be the location of the mobile device, thetime and the date when the product was used (i.e. when the product soundwas detected). The location of the mobile device may be determined usingthe GPS-capability of the mobile device itself. The time and date may belogged by the mobile device's processing unit on receipt of the signalfrom the microphone. In embodiments, the location and time/dateinformation of the product use event may be transmitted to a system runby the owner of the identified brand, in order to provide them withprecise information on the usage of their products. Such informationmay, for example, enable a brand owner to determine that one of theirproducts is typically used by consumers on weekdays at lunchtime.

In embodiments, the event data may be transmitted to a further system torequest additional content from an online service. Preferably, the eventdata is transmitted to a further system which is configured to delivercontent or additional functionality to the mobile device, i.e. a‘reward’. The reward may be, for example, access to an exclusivepromotional video linked to the brand, or a monetary reward such as apercentage discount off the user's next purchase of the branded product.The reward may be delivered on a conditional basis, such as when a userhas opened/consumed a specific number of the branded beverages. Thefurther system is configured to communicate with the mobile device'soperating system to deliver the reward in the appropriate manner (e.g.causing a video to be played or a money-off coupon to be downloaded).

Turning now to FIG. 2, this shows a process to detect brand sonificationvia the internet and cloud computing in an embodiment of the invention.Here, the local computing device (i.e. the user's mobile device)receives a sound for identification, but rather than perform the soundidentification analysis on the local processing unit, the sound data istransmitted for analysis to an external processing unit (e.g. to ‘thecloud’) via a wired or wireless communication channel. Currently, thecomputational power of many mobile devices limits the ability to performthe sound analysis on the local processing unit. Thus, advantageously,in the embodiment of FIG. 2 the sound analysis is performed via ‘thecloud’. The audio data transmitted to ‘the cloud’ for processing may bea coarser representation of the sound than the original audio, e.g. aFast Fourier Transform of the original data. This coarser representationis generated by the internal processing unit of the mobile device.

If the received audio data matches a sound model, event data isdelivered to a further system as previously outlined with reference toFIG. 1. During the analysis of the audio data, the external processingunit may store the audio data in an audio database. One or more soundmodels may be stored within ‘the cloud’ or elsewhere, such that thesound models are accessible to the external processing unit. Asmentioned above, the sound models may be updated based on received audiodata in order to improve the original models. Thus, the audio datastored within the audio database may be sent to a system for updatingthe sound models. Preferably, for efficiency, new audio data is onlytransmitted to the model updating system when a certain number of newaudio recordings have been received. Thus, preferably, audio recordingsare sent to the model updating system in batches (e.g. after a specificnumber of new audio recordings is received), and the sound models areupdated as described in section 4 below.

FIG. 3 illustrates a process to detect brand sonification on a mobiledevice in another embodiment of the invention. Here, the mobile deviceprocessor may have the processing capability required to perform thesound identification analysis. In FIG. 3, the local processing unit isshown to have access to locally stored sound models (i.e. stored on themobile device), but it may be possible that the sound models are storedexternally (e.g. in ‘the cloud’) and accessed by the local processingunit when perform the sound analysis. The local processing unit may, onsuccessful identification of brand sonification, create and deliver aninteractive experience for the user on the mobile device.

In FIG. 3, even though the local processing unit performs the soundanalysis, the local processing unit does not perform any updates to thesound models itself. This is because the sound models are ideallyupdated using audio data collected from multiple users, and the localprocessing unit does not have access to an audio database. Rather, thereceived audio data may be transmitted to an externally located server,audio database and model updating system, e.g. located in ‘the cloud’.Additionally or alternatively, the updating task may be distributed overa number of different systems, devices or processors, e.g. using atraining process distributed in ‘the cloud’. New models and updatedmodels created using the user audio data are sent back to user mobiledevices in order to enable the local processing unit to perform thesound analysis the next time a sound is received by the device.

As shown in FIG. 3, in one embodiment, the local processing unit mayitself create a local interactive experience for the user. Additionallyor alternatively, the local processing unit may transmit event data to afurther, external system which is configured to create and deliver aninteractive experience or reward for the user, as described withreference to FIG. 1. The event data may be delivered to a further systemrun by the brand owner, in order to provide them with preciseinformation on the usage of their products, as outlined above.

2. Product Modification

As described above, brand owners may be keen to identify when and wheretheir branded products are used or activated and the brand sonificationidentification process enables them to obtain this information. In theabove described embodiments, it may in certain circumstances bedifficult to determine the brand owner from the audio data alone. Forexample, many drinks manufacturers use similar or standard containersfor their drinks, which make the same or similar sounds when they areopened. In this situation, the user may be required to input furtherinformation into their mobile device before a ‘reward’ is delivered. Forinstance, once the sound identification process has determined that thesound represented the opening of a pressurized beverage can, the usermay be prompted to input the brand owner or select the brand owner froma list provided by the local processing unit. Thus, brand owners maywish to modify their product packaging in order to achieve brandsonification.

Turning now to FIG. 4a , this shows a series of pictures illustratingthe process of opening a typical pressurized beverage can 10 whichcomprises a ring pull 12 (also known as a pull-tab) attached at a rivetto the lid of the can. The lid is perforated or scored to define a tab14 in the lid. As shown in FIG. 4a , pulling the ring pull 12 causes thetab-end of the ring pull 12 to exert a downwards force on the tab 14,and eventually causes the tab 14 to be pushed into the can to create anopening. Many drinks manufacturers use the same ring pull design and usethe same can shape and size for their beverages, and thus, it can bedifficult to distinguish between brands based on the sound created uponpulling the ring pull and opening the can. However, brand owners may beable to achieve brand sonification by modifying the ring pull. Forexample, FIG. 4b shows a modified tab 14 on a can 10. The tab 14comprises one or more ridges 16 which can cause a specific, distinctivesound to be created when the ring pull is pulled to open the can. Thering pull 12 may modified (not shown) so that it interacts with theridges when pulled. For instance, the tab-end of the ring pull 12 may beextended so that it runs over the one or more ridges 16 when it isoperated in the usual way. Or, the ring pull and rivet may be configuredto enable the tab-end of the ring pull to slide over the ridges 16, suchthat as the ring pull is moved from the horizontal position (on theleft-hand side of FIG. 4a ) into the vertical/upright position (on theright-hand side of FIG. 4 a), the ring pull 12 is successively broughtinto contact with ridges 16 to cause a series of “clicks”. Suchmodifications may enable a specific sound to be created when the can isopened, which can then be more easily determined as brand sonificationby the sound analysis process.

3. Sound Identification

The applicant's PCT application WO2010/070314, which is incorporated byreference in its entirety, describes in detail various methods toidentify sounds. Broadly speaking an input sample sound is processed bydecomposition into frequency bands, and optionally de-correlated, forexample, using PCA/ICA, and then this data is compared to one or moreMarkov models to generate log likelihood ratio (LLR) data for the inputsound to be identified. A (hard) confidence threshold may then beemployed to determine whether or not a sound has been identified; if a“fit” is detected to two or more stored Markov models then preferablythe system picks the most probable. A sound is “fitted” to a model byeffectively comparing the sound to be identified with expected frequencydomain data predicted by the Markov model. False positives are reducedby correcting/updating means and variances in the model based oninterference (which includes background) noise.

There are several practical considerations when trying to detect soundsfrom compressed audio formats in a robust and scalable manner. Where thesound stream is uncompressed to PCM (pulse code modulated) format andthen passed to a classification system, the first stage of an audioanalysis system may be to perform a frequency analysis on the incominguncompressed PCM audio data. However, the recently compressed form ofthe audio may contain a detailed frequency description of the audio, forexample where the audio is stored as part of a lossy compression system.By directly utilising this frequency information in the compressed form,i.e., sub-band scanning in an embodiment of the above still furtheraspect, a considerable computational saving may be achieved by notuncompressing and then frequency analyzing the audio. This may mean asound can be detected with a significantly lower computationalrequirement. Further advantageously, this may make the application of asound detection system more scalable and enable it to operate on deviceswith limited computational power which other techniques could notoperate on.

The digital sound identification system may comprise discrete cosinetransform (DCT) or modified DCT coefficients. The compressed audio datastream may be an MPEG standard data stream, in particular an MPEG 4standard data stream.

The sound identification system may work with compressed audio oruncompressed audio. For example, the time-frequency matrix for a 44.1KHz signal might be a 1024 point FFT with a 512 overlap. This isapproximately a 20 milliseconds window with 10 millisecond overlap. Theresulting 512 frequency bins are then grouped into sub bands, or examplequarter-octave ranging between 62.5 to 8000 Hz giving 30 sub-bands.

A lookup table is used to map from the compressed or uncompressedfrequency bands to the new sub-band representation bands. For the samplerate and STFT size example given the array might comprise of a (Binsize÷2)×6 array for each sampling-rate/bin number pair supported. Therows correspond to the bin number (centre)—STFT size or number offrequency coefficients. The first two columns determine the lower andupper quarter octave bin index numbers. The following four columnsdetermine the proportion of the bins magnitude that should be placed inthe corresponding quarter octave bin starting from the lower quarteroctave defined in the first column to the upper quarter octave bindefined in the second column. e.g. if the bin overlaps two quarteroctave ranges the 3 and 4 columns will have proportional values that sumto 1 and the 5 and 6 columns will have zeros. If a bin overlaps morethan one sub-band more columns will have proportional magnitude values.This example models the critical bands in the human auditory system.This reduced time/frequency representation is then processed by thenormalization method outlined. This process is repeated for all framesincrementally moving the frame position by a hop size of 10 ms. Theoverlapping window (hop size not equal to window size) improves thetime-resolution of the system. This is taken as an adequaterepresentation of the frequencies of the signal which can be used tosummarise the perceptual characteristics of the sound. The normalizationstage then takes each frame in the sub-band decomposition and divides bythe square root of the average power in each sub-band. The average iscalculated as the total power in all frequency bands divided by thenumber of frequency bands. This normalised time frequency matrix is thepassed to the next section of the system where its mean, variances andtransitions can be generated to fully characterize the sound's frequencydistribution and temporal trends. The next stage of the soundcharacterization requires further definitions. A continuous hiddenMarkov model is used to obtain the mean, variance and transitions neededfor the model. A Markov model can be completely characterized by λ=(A,B, Π) where A is the state transition probability matrix, B is theobservation probability matrix and Π is the state initializationprobability matrix. In more formal terms:

A=└a _(ij)┘ where a _(ij) ≡P(q _(t+1) =S _(j) |q _(t) =S _(i))

B=└b _(j)(m)┘ where b _(j)(m)≡P(O _(t) =v _(m) |q _(t) =S _(j))

Π=[π_(i)] where π_(i) ≡P(q ₁ =S _(i))

where q is the state value, O is the observation value. A state in thismodel is actually the frequency distribution characterised by a set ofmean and variance data. However, the formal definitions for this will beintroduced later. Generating the model parameters is a matter ofmaximizing the probability of an observation sequence. The Baum-Welchalgorithm is an expectation maximization procedure that has been usedfor doing just that. It is an iterative algorithm where each iterationis made up of two parts, the expectation ε_(t)(i, j) and themaximization γ_(t)(i). In the expectation part, ε_(t)(i, j) andγ_(t)(i), are computed given λ, the current model values, and then inthe maximization λ is step recalculated. These two steps alternate untilconvergence occurs. It has been shown that during this alternationprocess, P(O|λ) never decreases. Assume indicator variables z_(i) ^(t)as

Expectation${ɛ_{t}( {i,j} )} = \frac{{\alpha_{t}(i)}a_{ij}{b_{j}( O_{t + 1} )}{\beta_{t + 1}(j)}}{\sum\limits_{k}{\sum\limits_{l}{{\alpha (k)}a_{kl}{b_{l}( O_{t + 1} )}{\beta_{t + 1}(l)}}}}$${\gamma_{t}(i)} = {\sum\limits_{j = 1}^{N}{ɛ_{t}( {i,j} )}}$E[z_(i)^(t)] = γ_(t)(i)  and  [z_(ij)^(t)] − ɛ_(t)(i, j)$z_{i}^{t} = \{ {{\begin{matrix}1 & {{{if}\mspace{14mu} q_{t}} = S_{i}} \\0 & {otherwise}\end{matrix}z_{ij}^{t}} = \{ {{\begin{matrix}1 & {{{if}\mspace{14mu} q_{t}} = {{S_{i}\mspace{14mu} {and}\mspace{14mu} q_{t + 1}} = S_{j}}} \\0 & {otherwise}\end{matrix}{Maximisation}{\hat{a}}_{ij}} = {{\frac{\sum\limits_{k = 1}^{K}{\sum\limits_{t = 1}^{T_{k} - 1}{ɛ_{t}^{k}( {i,j} )}}}{\sum\limits_{k = 1}^{K}{\sum\limits_{t = 1}^{T_{k} - 1}{\gamma_{t}^{k}(i)}}}{{\hat{b}}_{j}(m)}} = {{\frac{\sum\limits_{k = 1}^{K}{\sum\limits_{t = 1}^{T_{k} - 1}{{\gamma_{t}^{k}(j)}1( {O_{t}^{k} = v_{m}} )}}}{\sum\limits_{k = 1}^{K}{\sum\limits_{t = 1}^{T_{k} - 1}{\gamma_{t}^{k}(j)}}}\hat{\pi}} = \frac{\sum\limits_{K = 1}^{K}{\gamma_{1}^{k}(i)}}{K}}}} } $

Gaussian mixture models can be used to represent the continuousfrequency values, and expectation maximization equations can then bederived for the component parameters (with suitable regularization tokeep the number of parameters in check) and the mixture proportions.Assume a scalar continuous frequency value, O_(t)∈

with a normal distribution

p(O _(t) |q _(t) =S _(j), λ)˜N(μ_(j), σ_(j) ²)

This implies that in state S_(j), the frequency distribution is drawnfrom a normal distribution with mean μ_(j) and variance σ_(j) ². Themaximization step equation is then

${\hat{\mu}}_{j} = \frac{\sum\limits_{t}{{\gamma_{t}(j)}O_{t}}}{\sum\limits_{t}{\gamma_{t}(j)}}$${\hat{\sigma}}_{j}^{2} = \frac{\sum\limits_{t}{{\gamma_{t}(j)}( {O_{t - 1} - {\hat{\mu}}_{j}} )^{2}}}{\sum\limits_{t}{\gamma_{t}(j)}}$

The use of Gaussians enables the characterization of the time-frequencymatrix's features. In the case of a single Gaussian per state, theybecome the states. The transition matrix of the hidden Markov model canbe obtained using the Baum-Welch algorithm to characterize how thefrequency distribution of the signal change over time.

The Gaussians can be initialized using K-Means with the starting pointsfor the clusters being a random frequency distribution chosen fromsample data.

4. Matching New Sounds to Model(s)

To classify new sounds and adapt for changes in the acoustic conditions,a forward algorithm can be used to determine the most likely state pathof an observation sequence and produce a probability in terms of a loglikelihood that can be used to classify and incoming signal. The forwardand backward procedures can be used to obtain this value from thepreviously calculated model parameters. In fact only the forward part isneeded. The forward variable α_(t)(i) is defined as the probability ofobserving the partial sequence {O₁ . . . O_(t)} until time t and beingin S_(i) at time t, given the model λ.

α_(t)(i)≡P(O ₁ . . . O _(t) , q _(t) =S _(i)|λ)

This can be calculated by accumulating results and has two steps,initialization and recursion. α_(t)(i) explains the first t observationsand ends in state S_(i). This is multiplied by the probability α_(ij) ofmoving to state S_(j), and because there are N possible previous states,there is a need to sum over all such possible previous S_(i). The termb_(j)(O_(t+1)) is then the probability of generating the nextobservation, frequency distribution, while in state S_(j) at time t+1.With these variables it is then straightforward to calculate theprobability of a frequency distribution sequence.

${P( {O\lambda} )} = {\sum\limits_{i = 1}^{N}{\alpha_{T}(i)}}$

Computing α_(t)(i) has order O(N²T) and avoids complexity issues ofcalculating the probability of the sequence. The models will operate inmany different acoustic conditions and as it is practically restrictiveto present examples that are representative of all the acousticconditions the system will come in contact with, internal adjustment ofthe models will be performed to enable the system to operate in allthese different acoustic conditions. Many different methods can be usedfor this update. For example, the method may comprise taking an averagevalue for the sub-bands, e.g. the quarter octave frequency values forthe last T number of seconds. These averages are added to the modelvalues to update the internal model of the sound in that acousticenvironment.

No doubt many other effective alternatives will occur to the skilledperson. It will be understood that the invention is not limited to thedescribed embodiments and encompasses modifications apparent to thoseskilled in the art lying within the spirit and scope of the claimsappended hereto.

1. A mobile device comprising a software application configured to:detect the sound of a product use event; provide a user reward usingsaid software application in response to said detection; capture datarelating to said product use event; and provide said captured data to aremote computer system for analysis.
 2. A mobile device as claimed inclaim 1 wherein the sound of said product use event comprises the soundof opening a package of said product.
 3. A mobile device as claimed inclaim 2 wherein the sound of said product use event comprises the soundof a can ring-pull/tab opening event on a can of pressurized beverage.4. A mobile device as claimed in claim 2 wherein the sound of saidproduct use event comprises the sound of a screw cap twisting andopening event on a beverage bottle.
 5. A mobile device as claimed inclaim 1 wherein said captured data is location data of said mobiledevice when said sound is detected and/or date and time data. 6.(canceled)
 7. (canceled)
 8. A mobile device as claimed in claim 1wherein said user reward is only delivered when said softwareapplication has detected a specific number of product use events.
 9. Anon-transitory data carrier carrying processor control code for thesoftware application of claim
 1. 10. A method of capturing product usedata from users of a product, the method comprising: providing a mobiledevice with an app configured to identify use of said product from asound of said use and to provide an interactive user experience inresponse to said identification; and capturing use data of said productduring use of said app.
 11. A method of capturing product use data asclaimed in claim 10 further comprising modifying said product toincrease a distinctiveness of said product sound.
 12. A method ofcapturing product use data as claimed in claim 11 wherein a canring-pull/tab on a can of pressurized beverage is provided with ridges,grooves and/or indentations to increase said distinctiveness of saidproduct sound.
 13. A method of capturing product use data as claimed inclaim 12 wherein a screw cap on a beverage bottle is connected to a bandby a plurality of breakable thin connectors, and wherein said screw cap,said band and/or said plurality of thin connectors is modified toincrease said distinctiveness of said product sound when said screw capis twisted and said thin connectors are broken.
 14. A method as claimedin claim 10 wherein said mobile device has at least one microphone withaccess to audio analytic software via said app, configured to analyzeaudio data captured by said microphone to identify a sound associatedwith use of said product; using said mobile device to identify when saidproduct is used from said sound of use of said product; and capturingdata relating to said use of said product in response to saididentification.
 15. A method of capturing product use data from users ofa product, as claimed in claim 14, further comprising providing saidconsumer electronic device with an interactive software applicationhaving access to said audio analytic software, wherein said interactivesoftware application is configured to enable user interaction with saidconsumer electronic device in response to said sound identification. 16.(canceled)
 17. A method of capturing product use data from users of aproduct, as claimed in claim 14, wherein said audio analytic software isprovided on a network of remote servers hosted on the Internet, and saidcaptured audio data is transmitted to said remote servers for saidanalysis.
 18. A method of capturing product use data from users of aproduct, as claimed in claim 15, wherein said interactive softwareapplication has access to one or more sound models and wherein saidaudio analytic software compares said captured audio data with said oneor more sound models to identify said sound.
 19. A method of capturingproduct use data from users of a product, as claimed in claim 18 whereinsaid one or more sound models are updated and improved using said audiodata captured by said consumer electronic device.
 20. A method ofcapturing product use data from users of a product, as claimed in claim19, wherein said one or more sound models is provided on said network ofremote servers and said audio data captured from one or more consumerelectronic devices is transmitted to said remote servers to enable saidone or more sound models to be updated.
 21. A method of capturingproduct use data from users of a product, as claimed in claim 19 whereinif said audio analytic software is provided within said consumerelectronic device, said updated sound models are delivered to saidconsumer electronic device.
 22. (canceled)
 23. (canceled)
 24. A systemfor identifying a sound associated with a product use event, the systemcomprising: non-volatile memory for storing one or more sound models andfor storing processor control code; a sound data input; a processorcoupled to said sound data input and to said stored processor controlcode, wherein said processor control code comprises code to: input, fromsaid sound data input, sample sound data for said sound associated withsaid product use event to be identified; input event data associatedwith said sound data input; compare said sample sound data with saidstored one or more sound models; identify a product associated with saidsample sound data; and deliver an interactive user experience associatedwith said identification to a user of said product.
 25. A system foridentifying a sound associated with a product use event as claimed inclaim 24, wherein said sound data is captured by a consumer electronicdevice or a vending machine, and wherein said captured sound istransmitted by said consumer electronic device or said vending machineby a wired or wireless connection to said system for processing. 26.(canceled)